Skip to content

fix(bootstrap): install openclaw/acpx from a committed lockfile - #239

Merged
jsugg merged 4 commits into
mainfrom
fix/node-tools-lockfile
Nov 11, 2026
Merged

jsugg merged 4 commits into
mainfrom
fix/node-tools-lockfile

Conversation

@jsugg

@jsugg jsugg commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Root cause

clawops bootstrap ran npm install -g openclaw@2026.3.13 acpx@0.3.0 (sudo on Linux). Only the two direct packages were pinned. Neither tarball ships a shrinkwrap, so every bootstrap re-resolved about 570 transitive packages from live registry metadata. Run 42883800324, attempt 1, failed twice:

  1. ETARGET protobufjs@^7.5.5, although 7.5.5 was published 2026-04-15. The cause was stale cached metadata. fresh-host-core.yml set npm_config_prefer_offline=true, which skips packument revalidation, and restored a broad fresh-host-packages-<os>-<arch>- fallback cache. fresh-host-cache-warm.yml then re-saved that fallback content under new exact keys, so stale packuments carried forward indefinitely.
  2. ETARGET @aws-sdk/credential-provider-http@^3.972.86, after the npm cache clean --force retry. This was a live, non-atomic AWS SDK publish. credential-provider-node@3.972.96 (19:40:41Z) required the http package 29s before it was published (19:41:10Z). Cleaning the cache cannot fix this: a time-dependent resolver can always land inside such a window.

Design

  • Lock. platform/node-tools/{package.json,package-lock.json} is committed and mirrored to src/clawops/assets/platform/node-tools via sync_runtime_assets.py. It was generated with Node 22.22.1 / npm 10.9.4, and the versions are unchanged (openclaw 2026.3.13, acpx 0.3.0). 571 of 572 entries carry a registry resolved URL and sha512 integrity. The one exception is libsignal, a git dependency pinned to a 40-hex commit: npm records no integrity for git deps. The contract test allow-lists it explicitly.
  • Single source of truth. DEFAULT_OPENCLAW_VERSION and DEFAULT_ACPX_VERSION are read from the packaged manifest. platform_compat.py stays stdlib-only so the Makefile can still run it as a script. The varlock default and the memory-plugin helper reuse the constant. A contract test finds every tracked openclaw@X / OpenClaw `X` / OPENCLAW_VERSION=X / acpx@X / ACPX `X` mention and requires it to equal the manifest.
  • Install (clawops/strongclaw_node_tools.py):
    • Runs npm ci --omit=dev --no-audit --no-fund into <npm prefix -g>/lib/strongclaw-node-tools/.staging-<lock-digest>-<pid>.
    • Smoke-tests openclaw --version (prints OpenClaw 2026.3.13 (61d171a)) and acpx --version (0.3.0).
    • Renames the staging dir to <digest>, re-smokes it, then atomically switches the current symlink.
    • The previous release stays active until the new one passes.
    • A re-run with the same lock is a no-op (no mutations).
  • Bins. Linked as <npm prefix -g>/bin/<bin> -> ../lib/strongclaw-node-tools/current/node_modules/.bin/<bin>. This is exactly where npm install -g put them, and Linux keeps the same sudo rule. A foreign file or symlink at a bin path is a hard error, as npm install -g would also refuse.
  • Migration and legacy-layout compatibility.
    • Legacy bin links are swapped first, each with temp-symlink + rename(2). There is no window where a command is missing.
    • Then each <prefix>/lib/node_modules/<pkg> becomes a compat symlink to ../strongclaw-node-tools/current/node_modules/<pkg>.
      • Absolute-path consumers of the old global layout keep working, and they follow every upgrade switch.
      • A real legacy dir is moved aside into the tool root (pruned after success), then the symlink is created. rename(2) cannot swap a directory for a symlink, so for absolute-path readers only, there is a sub-millisecond gap between those two renames. Bins are unaffected.
    • The memory plugin's extensionAPI.js probe also lists the managed path directly, so it does not depend only on the compat link.
  • npm semantics with compat links (verified with real npm 10.9.4 and 11.x):
    • npm ls -g lists openclaw@2026.3.13 -> ./strongclaw-node-tools/<digest>/node_modules/openclaw (and acpx) as linked packages.
    • npm uninstall -g openclaw removes only that package's compat link and bin link. It never touches the managed release (it does not follow the link). This is an explicit uninstall, with the same semantics as today.
      • Guard: re-running clawops bootstrap detects the missing links and restores them (changed=True). doctor-host already fails with "Run clawops bootstrap" if openclaw/acpx is missing.
    • npm install -g openclaw@X over a compat link swaps in a floating install. The next bootstrap classifies it as replaceable and migrates back to the lock.
  • Retries. Only transient failures are retried: ECONNRESET, ETIMEDOUT, EAI_AGAIN, ECONNREFUSED, 5xx/429, and truncated tarballs. Backoff is 5s/15s/45s, 4 attempts max. ETARGET, E404, EINTEGRITY, EUSAGE and lifecycle failures fail immediately. The npm cache clean step is removed.
  • CI.
    • npm_config_prefer_offline is removed. The fresh-host cache is keyed on uv.lock + the node-tools lock + the memory-plugin lock, with no restore-keys.
    • The warm job probes with lookup-only and warms/saves only on a miss, starting from an empty cache. It now also prefetches the locked node-tools tarballs.
    • No other CI npm path needed prefer-offline; they all use npm ci against locks, except QMD's pinned npm install, which now just revalidates.
  • Updates. A Dependabot npm entry covers both directories, grouped by dependency so the mirror stays identical, with a 14-day cooldown. tests/scripts/update_node_tools_lock.py handles manual bumps: it checks for Node 22.22.1/npm 10.9.4, applies a 3-day --before release quarantine, validates, smoke-installs with npm ci, rewrites quoted versions, and syncs the mirror.

Legacy global-layout consumers (audited)

Runtime, absolute lib/node_modules path:

  • platform/plugins/memory-lancedb-pro/index.ts:385-393 (getExtensionApiImportSpecifiers): legacy /usr, /usr/local, /opt/homebrew lib/node_modules/openclaw/dist/extensionAPI.js fallbacks. These are served by the compat link, and the managed lib/strongclaw-node-tools/current/... path is now probed first.
    • Correction to the initial report: the plugin is installed with plain npm ci (strongclaw_bootstrap.py:402; the --omit=dev at :485 is lossless-claw). Its bare import fails because openclaw is not a dependency of the plugin at all.

Plugin tests, NODE_PATH. These now also list the hoisted <prefix>/lib/strongclaw-node-tools/current/node_modules, because npm ci hoists OpenClaw's deps instead of nesting them under the package:

  • test/cli-smoke.mjs:12-20, test/context-support-e2e.mjs:19-21, test/memory-update-supersede.test.mjs:18-20, test/memory-upgrader-diagnostics.test.mjs:8-10, test/plugin-manifest-regression.mjs:13-15, test/preference-slots.test.mjs:7-9, test/smart-extractor-branches.mjs:12-14, test/smart-metadata-v2.mjs:12-14, test/temporal-facts.test.mjs:12-14, test/vector-search-cosine.test.mjs:15-17 (all under platform/plugins/memory-lancedb-pro/)

Service units (PATH only, bin dirs unchanged):

  • platform/launchd/ai.openclaw.{backup-create,backup-verify,maintenance}.plist.template:34 and ai.openclaw.{browserlab,gateway,sidecars}.plist.template:39 (/usr/local/bin:/opt/homebrew/bin:/usr/bin)
  • platform/systemd/openclaw-{backup-create,backup-verify,gateway,maintenance}.service:15 and openclaw-{browserlab,sidecars}.service:16 (/usr/local/bin:/usr/bin)
  • tests/utils/helpers/_fresh_host/shell.py:77 (fresh-host phase PATH)

Doctor and other bare-name callers (resolve via PATH, unchanged):

  • src/clawops/setup_cli.py:129-131,153-156
  • strongclaw_runtime.py:858,881, strongclaw_ops.py:940, strongclaw_model_auth.py:393, acp_runner.py:258, strongclaw_recovery.py:209-211, recovery/backends.py:42-49, memory_tools.py:428

Other:

  • npm root -g / npm prefix -g lookups: none outside the new strongclaw_node_tools.py.
  • Docs: platform/docs/HOST_PLATFORMS.md:23-24, platform/workers/acpx/README.md:20, and memory-lancedb-pro/STRONGCLAW_VENDOR.md (local-patch notes).
  • Test mocks returning /usr/local/bin/openclaw or /usr/bin/openclaw (tests/suites/unit/ci/test_security_workflow.py:303,338-340, tests/suites/unit/clawops/recovery/test_backends.py:14, tests/suites/unit/clawops/test_strongclaw_recovery.py:66,330) only stub shutil.which; no change needed.

PATH compatibility proof

Every src/clawops call site resolves the CLIs by bare name on PATH: setup_cli doctor, strongclaw_runtime.run_openclaw_command, strongclaw_ops gateway, model_auth, recovery, acp_runner, and memory_tools. The launchd/systemd units hardcode PATH=...:/usr/local/bin:/opt/homebrew/bin:/usr/bin:.... Keeping the bins in $(npm prefix -g)/bin therefore keeps every one of them working.

Each OS run below did: baseline with today's npm install -g → new installer (migration) → legacy absolute-path import → npm ls -g → npm uninstall -g openclaw → bootstrap re-run (restore) → re-run (no-op).

macOS (Homebrew node 26.6.0 / npm 11.18.0, prefix /usr/local), probed with env -i HOME=$HOME /bin/zsh -l and /bin/bash -l:

baseline   openclaw -> /usr/local/bin/openclaw | OpenClaw 2026.3.13 (61d171a)   acpx -> /usr/local/bin/acpx | 0.3.0
migrated   /usr/local/bin/openclaw -> ../lib/strongclaw-node-tools/current/node_modules/.bin/openclaw
           /usr/local/lib/node_modules/openclaw -> ../strongclaw-node-tools/current/node_modules/openclaw   (acpx likewise)
           zsh -l / bash -l: openclaw -> /usr/local/bin/openclaw | OpenClaw 2026.3.13 ; acpx -> /usr/local/bin/acpx | 0.3.0
legacy     import('file:///usr/local/lib/node_modules/openclaw/dist/extensionAPI.js').runEmbeddedPiAgent: function
npm ls -g  openclaw@2026.3.13 -> ./strongclaw-node-tools/483b32af6b1e16b2/node_modules/openclaw   (acpx likewise)
uninstall  npm uninstall -g openclaw -> openclaw MISSING, acpx still resolves, managed release intact
restore    bootstrap re-run: RESULT changed=True -> both resolve again
re-run     RESULT changed=False

Linux (ubuntu:24.04 container, NodeSource Node 22 / npm 10.9.9 as bootstrap installs it, non-root sudo user, sudo npm prefix -g = /usr), probed with env -i HOME=... /bin/bash -l:

baseline   openclaw -> /usr/bin/openclaw | OpenClaw 2026.3.13 (61d171a)   acpx -> /usr/bin/acpx | 0.3.0
migrated   root-owned /usr/bin/openclaw -> ../lib/strongclaw-node-tools/current/node_modules/.bin/openclaw
           root-owned /usr/lib/node_modules/openclaw -> ../strongclaw-node-tools/current/node_modules/openclaw   (acpx likewise)
           dev login shell and a separate unprivileged runtime user (svc): both resolve /usr/bin/{openclaw,acpx}
legacy     import('file:///usr/lib/node_modules/openclaw/dist/extensionAPI.js').runEmbeddedPiAgent: function
npm ls -g  openclaw@2026.3.13 -> ./strongclaw-node-tools/483b32af6b1e16b2/node_modules/openclaw   (acpx likewise)
uninstall  sudo npm uninstall -g openclaw -> openclaw MISSING, acpx still resolves, managed release intact
restore    bootstrap re-run: RESULT changed=True -> both resolve again
re-run     RESULT changed=False

Test evidence

  • Unit (test_strongclaw_node_tools.py, 36 tests). A real-filesystem fake npm covers:
    • npm ci is used and never install -g
    • staging, atomic switch, and the no-op re-run
    • legacy migration ordering (bins swapped before any compat-link change, no remove of a bin path)
    • compat links exist and resolve into the active release after install, upgrade switch, failed upgrade, and no-op re-run
    • the legacy lib/node_modules/openclaw/dist/extensionAPI.js path follows the upgrade switch
    • foreign-bin collisions
    • a failed smoke test keeps the previous release
    • lock change + prune, and rebuild of an unhealthy release
    • retry classification (ETARGET is never retried) and the backoff budget
    • sudo argv
    • rejection of locks missing resolved/integrity, unpinned git deps, links, and drift
  • Integration (test_strongclaw_node_tools_npm.py, real npm + node, passes on npm 10.9.4 and 11.x):
    • real node imports extensionAPI.js through the legacy path, and npm ls -g lists the links
    • npm uninstall -g only unlinks and bootstrap restores
    • npm install -g over a compat link is reconciled back
  • Contracts:
    • the bundle is valid in both copies, and constants derive from the manifest
    • lock integrity (registry sha512 plus the reviewed git allow-list)
    • quoted versions match everywhere
    • package-data contains the bundle, and the wheel ships both files
    • no workflow sets prefer-offline
    • the cache key is identical and lock-based with no restore-keys
    • the warm job is lookup-only and miss-gated
    • the Dependabot npm entry exists
  • Updater: unit tests, plus a real run with Node 22.22.1/npm 10.9.4 that regenerates the committed lock byte-for-byte (including the 3-day quarantine and npm ci smoke).
  • Local gates: make lint imports typecheck (ruff, isort, black, mypy, pyright: clean), make test-contracts (134 passed), make test (1352 passed; test_hosted_docker_pull_client process-group test flaked once under load, passes 3/3 in isolation, untouched here), pre-commit actionlint/check-yaml/shellcheck (pass).

Notes

  • git stays a host prerequisite (the libsignal git dep), exactly as it already was for npm install -g.
  • memory-plugin-verification.yml still does a floating npm install openclaw@2026.3.13 into a temp dir. Its version quote is now contract-checked, but the install itself is not yet lock-based.

`npm install -g openclaw@2026.3.13 acpx@0.3.0` pinned only the two direct
packages, so every bootstrap re-resolved ~570 transitive packages from live
registry metadata. That broke on stale cached packuments (ETARGET
protobufjs@^7.5.5) and on a non-atomic AWS SDK publish (ETARGET
@aws-sdk/credential-provider-http@^3.972.86).

- Commit platform/node-tools/{package.json,package-lock.json} (Node 22.22.1,
  npm 10.9.4), mirrored into the packaged assets. Versions are unchanged.
- The manifest is the single source of truth: DEFAULT_OPENCLAW_VERSION and
  DEFAULT_ACPX_VERSION derive from it; the varlock default and memory-plugin
  helper reuse the constant; a contract test keeps docs/env/workflow quotes
  in sync.
- Bootstrap runs `npm ci --omit=dev --no-audit --no-fund` into a staging dir
  under <npm prefix -g>/lib/strongclaw-node-tools, smoke-tests both CLIs,
  then atomically switches a `current` symlink. Bins stay in
  <npm prefix -g>/bin with the same sudo rule on Linux, so PATH resolution is
  unchanged. Legacy global packages are deleted after the links are swapped,
  so `npm uninstall -g` can no longer remove them. A re-run with the same
  lock is a no-op.
- Retry only transient network/tarball failures with bounded backoff; drop
  the `npm cache clean --force` retry. ETARGET and other resolution errors
  fail immediately.
- Add tests/scripts/update_node_tools_lock.py for manual, quarantined bumps.
…allbacks

- Drop `npm_config_prefer_offline`: it skipped packument revalidation, so a
  months-old cached packument produced ETARGET protobufjs@^7.5.5. Nothing
  else needs it; every other CI npm path is `npm ci` against a lock.
- Key the fresh-host package cache on uv.lock plus the node-tools and
  memory-plugin locks, and remove the broad OS/arch restore-keys fallback.
- Cache warm: probe the exact key with `lookup-only` and warm/save only on a
  miss, from an empty cache, so old fallback content is never re-saved under
  a new key. The warmer now prefetches the locked node-tools tarballs.
- Dependabot: add an npm entry for platform/node-tools and its packaged
  mirror, grouped by dependency so both copies move together.
Deleting <prefix>/lib/node_modules/{openclaw,acpx} after migration broke
absolute-path consumers of the old `npm install -g` layout, notably the
memory plugin's openclaw/dist/extensionAPI.js fallbacks.

- After the bin links switch, replace each legacy package path with a
  symlink to ../strongclaw-node-tools/current/node_modules/<pkg>, so
  absolute-path readers follow every release switch. A real legacy dir is
  moved aside into the tool root and pruned after success.
- npm treats the compat links as linked globals: `npm ls -g` lists them,
  `npm uninstall -g <pkg>` removes only that package's links (never the
  release), and `npm install -g` swaps in a floating copy; the next
  bootstrap run reconciles both back to the lock.
- memory-lancedb-pro: probe the managed extensionAPI.js path ahead of each
  legacy path, and add the hoisted managed node_modules to the test
  NODE_PATH lists.
@jsugg
jsugg merged commit 99b0be4 into main Nov 11, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant